Project - Feature Selection, Model Selection and Tuning¶
Problem Statement: Concrete Strength Prediction¶
Objective¶
To predict the concrete strength using the data available in file "concrete.csv". Apply feature engineering and model tuning to obtain a score above 85%.¶
Steps and Tasks:¶
- Exploratory Data Quality Report Reflecting the Following:¶
- Univariate analysis –data types and description of the independent attributes which should include (name, range of values observed, central values (mean and median), standard deviation and quartiles, analysis of the body of distributions/tails, missing values, outliers, duplicates(10 Marks).
- Bi-variate analysis between the predictor variables and between the predictor variables and target column. Comment on your findings in terms of their relationship and degree of relation if any. Visualize the analysis using boxplots and pair plots, histograms, or density curves. (10 marks).
Feature Engineering techniques(10 marks):
3.1. Identify opportunities (if any) to extract new features from existing features, drop a feature(if required) Hint: Feature Extraction, for example, consider a dataset with two features length and breadth. From this, we can extract a new feature Area which would be length * breadth.
3.2 Get the data model ready and do a train test split.
3.3 Decide on the complexity of the model, should it be a simple linear model in terms of parameters or would a quadratic or higher degree.
- Creating the Model and Tuning It:¶
- Algorithms that you think will be suitable for this project. Use Kfold Cross-Validation to evaluate model performance. Use appropriate metrics and make a DataFrame to compare models w.r.t their metrics. (at least 3 algorithms, one bagging and one boosting based algorithms have to be there). (15 marks).
- Techniques employed to squeeze that extra performance out of the model without making it overfit. Use Grid Search or Random Search on any of the two models used above. Make a DataFrame to compare models after hyperparameter tuning and their metrics as above. (15 marks).
Attribute Information:¶
Given are the variable name, variable type, the measurement unit, and a brief description. The concrete compressive strength is the regression problem. The order of this listing corresponds to the order of numerals along the rows of the database.
Input variables:¶
Name -- Data Type -- Measurement -- Description
- Cement (cement) -- quantitative -- kg in a m3 mixture -- Input Variable
- Blast Furnace Slag (slag) -- quantitative -- kg in a m3 mixture -- Input Variable
- Fly Ash (ash) -- quantitative -- kg in a m3 mixture -- Input Variable
- Water(water) -- quantitative -- kg in a m3 mixture -- Input Variable
- Superplasticizer (superplastic)-quantitative -- kg in a m3 mixture -- Input Variable
- Coarse Aggregate (coarseagg) -- quantitative -- kg in a m3 mixture -- Input Variable
- Fine Aggregate (fineagg) -- quantitative -- kg in a m3 mixture -- Input Variable
- Age(age) -- quantitative -- Day (1~365) -- Input Variable
Output variable (desired target):¶
- Concrete compressive strength(strength) -- quantitative -- MPa -- Output Variable
0. Introduction¶
For this project I will use the experience from previous project as a template. This will allows me to move faster in the initial part of the project until I create a Base model. After this I will apply changes for tunning the model and get better results. Hence most of the intial steps (plots, graphics, functions,etc) will be similar to the one used in previous projects.¶
In previous project I was not able to apply modification due to limit of time, but as seen in the mentored session, applying a templete is a good strategy to be more efficient and get better results.¶
Having a base model with minimal modification in the data set will let us have a reference line to compare later on. Additionally we can come back and improve the Engineering and run the model again but the output will be a different model name¶
For this Project i will try to apply more structural steps, it is to say, separete the plot section from graphics, in order to move easily along the document. In previous projects after many lines it was difficult to move along the project.¶
1. Exploratory data quality report¶
Import Libraries¶
In [1]:
import warnings
warnings.filterwarnings('ignore')
In [2]:
import pandas as pd #Read files
import numpy as np # numerical libraries
# Import visualization libraries
import seaborn as sns
import matplotlib.pyplot as plt
#%matplotlib inline
from sklearn import metrics
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier
from sklearn import tree
from IPython.display import Image
from os import system
In [36]:
pd.options.display.float_format = '{:,.4f}'.format
In [4]:
# Below we will read the data from the local folder
df = pd.read_csv("concrete.csv")
# Now display the header
print ('concrete.csv data set:')
df.head(10)
Out[4]:
In [6]:
df.tail() ## to know how the end of the data looks like
Out[6]:
In [12]:
df.info() # here we will see the number of entires(rows and columns), dtype, and non-nullcount
In [22]:
print(f"The given dataset contains {df.shape[0]} rows and {df.shape[1]} columns")
print(f"The given dataset contains {df.isna().sum().sum()} Null value")
In [13]:
df.shape # size of the data set also shown in the cell above
Out[13]:
In [15]:
neg_exp=df[df.lt(0)] # this is to see the number of negative values present
print (" the number of negative entries is",len(neg_exp.index))
# the output might be taken in consideration later on in the calculations.
In [16]:
df.describe().transpose() # Transpose is used here to read better the attribute
Out[16]:
In [17]:
df.nunique() # Number of unique values in a column
# this help to identify categorical values.
Out[17]:
In [20]:
# Now we will get a list of unique values to evalaute how to arrange the data set
for a in list(df.columns):
n = df[a].unique()
# if number of unique values is less than 30, print the values. Otherwise print the number of unique values
if len(n)<30:
print(a + ': ')
print(df[a].value_counts(normalize=True))
print()
else:
print(a + ': ' +str(len(n)) + ' unique values')
print()
In [11]:
plt.subplots(figsize=(20, 20))
ax = sns.boxplot(data=df)
ax.set_xticklabels(ax.get_xticklabels(), rotation=45);
- Age
As expected (Insight 2) is showing more amont of outliers and also less outliers in other columns:Slag, water, superplastic and fineagg the pdays column seems to have continue values as seen in the boxplot and maybe will no add any value to the calculation.these boxplot will be evalauted with histograms below
Here I will check each variable to evaluate the body of distributions/tails¶
In [40]:
df.hist(stacked=False, bins=50, figsize=(30,30), layout=(3,3)); # Histogram of the tentative continous variable
# ***Please notice that some of these variable can be dropped after the graphical evaluation.
In [52]:
## Please notice: I found this commands on internet and it allows a better and faster visualitzation of multiples displot,
## compared with how i did it in the previous project and with the line above.
#### With this code we can also see the mean of each variable which is better than the histogram plotted above.
import itertools
import statistics
cols = [i for i in df.columns]
fig = plt.figure(figsize=(20, 25))
for i,j in itertools.zip_longest(cols, range(len(cols))):
plt.subplot(5,2,j+1)
ax = sns.distplot(df[i],color='blue',rug=True)
plt.axvline(df[i].mean(),linestyle="dashed",label="mean", color='black')
plt.axvline(statistics.mode(df[i]),linestyle="dashed",label="Mode", color='Red')
plt.axvline(statistics.median(df[i]),linestyle="dashed",label="MEDIAN", color='Green')
plt.legend()
plt.title(i)
plt.xlabel("")
Here I will check the correlation between variables¶
In [10]:
df.corr() # with this function will try to see a correlation between variables numerically
Out[10]:
- Strength
has higher correlation (>= 30%) with Cement,Superplastic, age and 29% with water(negative correlation) - Strenght
has a minor correlation (<1%) with the rest of variables. I will try to test a model without slag and ash since they seems to have low impact in the target variable- Cement and ash
shows a good negative correlation ~ -40% - Water and superplastic
have a strong negative correlation of -65%
In [56]:
g = sns.PairGrid(df)
g.map_upper(plt.scatter)
g.map_lower(sns.lineplot)
g.map_diag(sns.kdeplot, lw=3, legend=True);
In [8]:
sns.pairplot(df , hue='age' , diag_kind = 'kde')
plt.show()
- In the 1st Pair plot above:
here we see, in the diagonal, a almost normal distribution for cement, water, coarseagg,fineagg and strngthAlso we see a double gaussian (explained in the lectures) + skewness ( insight #5) in slag, ash, superlastic and agewe can also confirm graphically the observations in the Insight #6 about the correlation beteween the target variable and between other variables, however I found it easier to read the number than this graphics
- In the 2nd paiplot:
Here we see that we actually have different gaussians in the diagonal for every variable if we split the age. Hence it will be interesting to change the age variable in smaller groups is evaluate the results later onin this pairplot:Ash, superplastic and slagvariable show a more clear the double gaussian
In [23]:
#Another correlation methods
plt.figure(figsize=(10,10))
mask = np.zeros_like(df.corr('spearman'))
mask[np.triu_indices_from(mask)] = True
ax =sns.heatmap(df.corr(),
annot=True,
linewidths=.5,
center=0,
cmap="YlGnBu",
mask= mask,
)
ax.set_xticklabels(ax.get_xticklabels(), rotation=45);
plt.show()
With this plot it is easy to recongnise the findings observed in the Insight #6 where it takes longer to find the correlation between the variablesAdditional to Insights #6, here we also find correlation between other variables positive and negatively but less than 30%. this plot can be reviewed later on to determine the relevance of some variablesIt is confirmed that the 3 most important variables for the target are:cement, superplastic and age
- ### I will run Pandas profiling just to confirm what mentioned in the previous lines. This code was seen in the latest mentor session after all the exploration analisys was done but I want to confirm the findings so it can be used for futures projects to save some time. Also this step can be added as a common practice.
In [7]:
#pip install pandas-profiling[notebook]
In [152]:
from pandas_profiling import ProfileReport
profile = ProfileReport(df)
profile
Out[152]:
The main takeaway of this step is that for the next projects I will apply it at the beggning of the study. It give an idea of where to look at to see more details, like relevant correlations and more important variablesMost of the observations from this report about the data were mentioned already
2. Feature Engineering Tecniques¶
- No new features sin all the variables have the same unit (kg/cc)
- the outliers will be treated in the following step (2.3.b)
- We will drop ash', 'coarseagg', 'fineagg' since they have low correlation
- Age will be regropued to treat outliers in the step 2.3.b
- The reason for the double gaussian (Insight # 7) might be due to the presence of outliers and Age, as mentioned before this will be trated in 2.3.b
In [5]:
X = df.drop('strength', axis=1) # Seperating the target and the rest
Y = df['strength']
In [6]:
#from sklearn.model_selection import train_test_split # Splitting the data for training and testing out model
##Split into training and test set
x_train, x_test,y_train, y_test = train_test_split(X, Y, test_size=0.3, random_state=1) # 1 is just any random seed number
print (x_train.shape, x_test.shape)
x_train.head() # this is to review the columns
Out[6]:
In [15]:
print("{0:0.2f}% data is in training set".format((len(x_train)/len(df.index)) * 100))
print("{0:0.2f}% data is in test set".format((len(x_test)/len(df.index)) * 100))
In [16]:
x_train.dtypes
Out[16]:
I will try to print accuracy on test data using below models:¶
- Linear Regression
- Linear Regression with Polynomial features of degree 2
- Linear Regression with Polynomial features of degree 3
- Decision tree
- Random forest
- Lasso
- Lasso with polynomial features of degree 2
- Lasso with polynomial features of degree 3
- Ada boosting
- Gradient boosting
- KNN
- Support Vector machines
In [22]:
from sklearn.linear_model import LinearRegression, LogisticRegression,Ridge, Lasso
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import PolynomialFeatures
from sklearn.tree import DecisionTreeRegressor
from sklearn.ensemble import RandomForestRegressor, RandomForestClassifier, AdaBoostRegressor, GradientBoostingRegressor
from sklearn.neighbors import KNeighborsRegressor
from yellowbrick.classifier import ClassificationReport, ROCAUC
from sklearn.svm import SVR
In [51]:
from sklearn.metrics import roc_auc_score, accuracy_score
from sklearn.metrics import roc_curve
from sklearn.model_selection import KFold
from sklearn.model_selection import cross_val_score
The next commands were the 1st trial to get the score of all the models at once. Later on these code improved.¶
In [39]:
#Linear Regression
linR = LinearRegression()
linR.fit(x_train, y_train)
pred = linR.predict(x_test) # Predictions from linear regression
score0_train = linR.score(x_train, y_train)
score0_test = linR.score(x_test, y_test)
#Linear Regression with Polynomial features of degree 2
pipeline= Pipeline ([('poly', PolynomialFeatures(degree=2)),('reg',LinearRegression())])
pipeline.fit(x_train,y_train)
score1_train = pipeline.score(x_train,y_train) # Predictions from linear regression degree 2
score1_test= pipeline.score(x_test,y_test)
print('Score Linear Regression degree 2_train', score1_train)
print('Score Linear Regression degree 2_test', score1_test)
#Linear Regression with Polynomial features of degree 2
pipeline= Pipeline ([('poly', PolynomialFeatures(degree=3)),('reg',LinearRegression())])
pipeline.fit(x_train,y_train)
score2_train = pipeline.score(x_train,y_train) # Predictions from linear regression degree 2
score2_test= pipeline.score(x_test,y_test)
print('Score Linear Regression degree 3_train', score1_train)
print('Score Linear Regression degree 3_test', score1_test)
#dt = DecisionTree Regressor
dt = DecisionTreeRegressor(random_state=42, max_depth=4) #scale=True)
dt.fit(x_train, y_train)
score3_train= dt.score(x_train, y_train)
score3_test= dt.score(x_test, y_test)
pred_dt = dt.predict(x_test)
#dt = DecisionTree Regressor degree 2
pipeline= Pipeline ([('poly', PolynomialFeatures(degree=2)),('reg',DecisionTreeRegressor(random_state=7))])
pipeline.fit(x_train,y_train)
score4_train = pipeline.score(x_train,y_train)
score4_test= pipeline.score(x_test,y_test)
print('Score Dt Regressor degree 2_train', score1_train)
print('Score Dt Regressor degree 2_train', score1_test)
rf = RandomForestRegressor(random_state=42, max_depth=4)
rf.fit(x_train, y_train)
score5_train = rf.score(x_train, y_train)
score5_test= rf.score(x_test, y_test)
ls=Lasso(random_state=42)
ls.fit(x_train, y_train)
pred_ls = ls.predict(x_test)
score6_train=ls.score(x_train, y_train)
score6_test= ls.score(x_test, y_test)
print (score0_train, score1_train, score2_train, score3_train, score4_train, score5_train, score6_train)
print (score0_test, score1_test, score2_test, score3_test, score4_test, score5_test, score6_test)
The next command is the improvement of the line above, it is a little more efficient and it shows the table at the end, however it looks like it can be done with a loop.¶
In [67]:
num_folds = 50
seed = 7
kfold = KFold(n_splits=num_folds, random_state=seed)
#Linear Regression
model = LinearRegression()
model.fit(x_train, y_train)
cv = cross_val_score(model, X, Y, cv=kfold)
mean = round(cv.mean(), 3)
std = round(cv.std(), 3)
results = pd.DataFrame({'Model':['LinearRegression'],
'score_training': round(model.score(x_train, y_train),3),
'score_test':round(model.score(x_test, y_test),3),
'k_fold_mean':mean,
'k_fold_std':std,
'95% confidence intervals': str(round(mean-(1.96*std),3)) + ' <-> ' + str(round(mean+(1.96*std),3))
})
results = results[['Model', 'score_training','score_test','k_fold_mean', 'k_fold_std', '95% confidence intervals']]
#Linear Regression with Polynomial features of degree 2
model= Pipeline ([('poly', PolynomialFeatures(degree=2)),('reg',LinearRegression())])
model.fit(x_train, y_train)
cv = cross_val_score(model, X, Y, cv=kfold)
mean = round(cv.mean(), 3)
std = round(cv.std(), 3)
tempresults = pd.DataFrame({'Model':['LinearRegression degree 2'],
'score_training': round(model.score(x_train, y_train),3),
'score_test':round(model.score(x_test, y_test),3),
'k_fold_mean':mean,
'k_fold_std':std,
'95% confidence intervals': str(round(mean-(1.96*std),3)) + ' <-> ' + str(round(mean+(1.96*std),3))
})
results = pd.concat([results, tempresults])
#Linear Regression with Polynomial features of degree 3
model= Pipeline ([('poly', PolynomialFeatures(degree=3)),('reg',LinearRegression())])
model.fit(x_train, y_train)
cv = cross_val_score(model, X, Y, cv=kfold)
mean = round(cv.mean(), 3)
std = round(cv.std(), 3)
tempresults = pd.DataFrame({'Model':['LinearRegression degree 3'],
'score_training': round(model.score(x_train, y_train),3),
'score_test':round(model.score(x_test, y_test),3),
'k_fold_mean':mean,
'k_fold_std':std,
'95% confidence intervals': str(round(mean-(1.96*std),3)) + ' <-> ' + str(round(mean+(1.96*std),3))
})
results = pd.concat([results, tempresults])
#dt = DecisionTree Regressor
model = DecisionTreeRegressor(random_state=42, max_depth=4) #scale=True)
model.fit(x_train, y_train)
cv = cross_val_score(model, X, Y, cv=kfold)
mean = round(cv.mean(), 3)
std = round(cv.std(), 3)
tempresults = pd.DataFrame({'Model':['DecisionTree Regressor'],
'score_training': round(model.score(x_train, y_train),3),
'score_test':round(model.score(x_test, y_test),3),
'k_fold_mean':mean,
'k_fold_std':std,
'95% confidence intervals': str(round(mean-(1.96*std),3)) + ' <-> ' + str(round(mean+(1.96*std),3))
})
results = pd.concat([results, tempresults])
#dt = DecisionTree Regressor degree 2
model= Pipeline ([('poly', PolynomialFeatures(degree=2)),('reg',DecisionTreeRegressor(random_state=7))])
model.fit(x_train, y_train)
cv = cross_val_score(model, X, Y, cv=kfold)
mean = round(cv.mean(), 3)
std = round(cv.std(), 3)
tempresults = pd.DataFrame({'Model':['DecisionTree Regressor degree 2'],
'score_training': round(model.score(x_train, y_train),3),
'score_test':round(model.score(x_test, y_test),3),
'k_fold_mean':mean,
'k_fold_std':std,
'95% confidence intervals': str(round(mean-(1.96*std),3)) + ' <-> ' + str(round(mean+(1.96*std),3))
})
results = pd.concat([results, tempresults])
# RandomForestRegressor
model = RandomForestRegressor(random_state=42, max_depth=4)
model.fit(x_train, y_train)
cv = cross_val_score(model, X, Y, cv=kfold)
mean = round(cv.mean(), 3)
std = round(cv.std(), 3)
tempresults = pd.DataFrame({'Model':['Random Forest Regressor'],
'score_training': round(model.score(x_train, y_train),3),
'score_test':round(model.score(x_test, y_test),3),
'k_fold_mean':mean,
'k_fold_std':std,
'95% confidence intervals': str(round(mean-(1.96*std),3)) + ' <-> ' + str(round(mean+(1.96*std),3))
})
results = pd.concat([results, tempresults])
#Lasso
model=Lasso(random_state=42)
model.fit(x_train, y_train)
cv = cross_val_score(model, X, Y, cv=kfold)
mean = round(cv.mean(), 3)
std = round(cv.std(), 3)
tempresults = pd.DataFrame({'Model':['Lasso'],
'score_training': round(model.score(x_train, y_train),3),
'score_test':round(model.score(x_test, y_test),3),
'k_fold_mean':mean,
'k_fold_std':std,
'95% confidence intervals': str(round(mean-(1.96*std),3)) + ' <-> ' + str(round(mean+(1.96*std),3))
})
results = pd.concat([results, tempresults])
# Lasso with polynomial features of degree 2
model= Pipeline ([('poly', PolynomialFeatures(degree=2)),('reg',Lasso(random_state=42))])
model.fit(x_train, y_train)
cv = cross_val_score(model, X, Y, cv=kfold)
mean = round(cv.mean(), 3)
std = round(cv.std(), 3)
tempresults = pd.DataFrame({'Model':['Lasso degree 2'],
'score_training': round(model.score(x_train, y_train),3),
'score_test':round(model.score(x_test, y_test),3),
'k_fold_mean':mean,
'k_fold_std':std,
'95% confidence intervals': str(round(mean-(1.96*std),3)) + ' <-> ' + str(round(mean+(1.96*std),3))
})
results = pd.concat([results, tempresults])
# Lasso with polynomial features of degree 3
model= Pipeline ([('poly', PolynomialFeatures(degree=3)),('reg',Lasso(random_state=42))])
model.fit(x_train, y_train)
cv = cross_val_score(model, X, Y, cv=kfold)
mean = round(cv.mean(), 3)
std = round(cv.std(), 3)
tempresults = pd.DataFrame({'Model':['Lasso degree 3'],
'score_training': round(model.score(x_train, y_train),3),
'score_test':round(model.score(x_test, y_test),3),
'k_fold_mean':mean,
'k_fold_std':std,
'95% confidence intervals': str(round(mean-(1.96*std),3)) + ' <-> ' + str(round(mean+(1.96*std),3))
})
results = pd.concat([results, tempresults])
# Ada boosting
model = AdaBoostRegressor(n_estimators = 100, learning_rate=0.1, random_state=22)
model.fit(x_train, y_train)
cv = cross_val_score(model, X, Y, cv=kfold)
mean = round(cv.mean(), 3)
std = round(cv.std(), 3)
tempresults = pd.DataFrame({'Model':['Ada boosting'],
'score_training': round(model.score(x_train, y_train),3),
'score_test':round(model.score(x_test, y_test),3),
'k_fold_mean':mean,
'k_fold_std':std,
'95% confidence intervals': str(round(mean-(1.96*std),3)) + ' <-> ' + str(round(mean+(1.96*std),3))
})
results = pd.concat([results, tempresults])
# Gradient boosting
model = GradientBoostingRegressor(random_state=42, max_depth=4)
model.fit(x_train, y_train)
cv = cross_val_score(model, X, Y, cv=kfold)
mean = round(cv.mean(), 3)
std = round(cv.std(), 3)
tempresults = pd.DataFrame({'Model':['Gradient boosting'],
'score_training': round(model.score(x_train, y_train),3),
'score_test':round(model.score(x_test, y_test),3),
'k_fold_mean':mean,
'k_fold_std':std,
'95% confidence intervals': str(round(mean-(1.96*std),3)) + ' <-> ' + str(round(mean+(1.96*std),3))
})
results = pd.concat([results, tempresults])
## The next lines are to select the best model based on the k_fold_mean and add it in the last row
tmp_best = results.sort_values(['k_fold_mean'], ascending=False).head(1)
tmp_best['Model'] = 'Best Model = ' + tmp_best['Model']
results = results.append(tmp_best, ignore_index=True)
## This is the table with the scoring result of the algorithm with all the data
results
Out[67]:
In [93]:
#1st I will try to count the outliers in the variables as seen in the previous mentored sessions.
#Age was showing in the EDA the bigger ammount of outliers
q1= df.quantile(0.25)
q3= df.quantile(0.75)
IQR = q3-q1
low = q1-1.5*IQR #acceptable range
high = q3+1.5*IQR #acceptable range
outliers = pd.DataFrame(((df > (high)) | (df < (low))).sum(axis=0), columns=['Total of outliers'])
outliers['% equivalent'] = round(outliers['Total of outliers']*100/len(df), 3)
outliers
Out[93]:
In [123]:
q1 = df['age'].quantile(0.25) #first quartile value
q3 = df['age'].quantile(0.75) # third quartile value
iqr = q3-q1 #Interquartile range
low = q1-1.5*iqr #acceptable range
high = q3+1.5*iqr #acceptable range
df_in = df.loc[(df['age'] >= low) & (df['age'] <= high)] # meeting the acceptable range
df_out = df.loc[(df['age'] < low) | (df['age'] > high)] # not meeting the acceptable range
age_mean=int(df_in.age.mean()) #finding the mean of the acceptable range
print('age_mean =', '' ,age_mean)
print('Shape of df without outliers = ' ,df_in.shape)
print('# of outliers =', df_out.shape)
print('lower rage = ' , low ,'&', ' Hiher rage =' , high)
#imputing outlier values with mean value
df_out.age=age_mean
#getting back the original shape of df
df_rev=pd.concat([df_in,df_out]) #concatenating both dfs to get the original shape
df_rev.shape
print('Shape of original data frame', '' ,df.shape) # original data frame
print('Shape new df ' , ' ' , df_rev.shape) #new df
In [124]:
## The code from above is repeated to check the outliers again
q1= df_rev.quantile(0.25)
q3= df_rev.quantile(0.75)
IQR = q3-q1
low = q1-1.5*IQR #acceptable range
high = q3+1.5*IQR #acceptable range
outliers = pd.DataFrame(((df_rev > (high)) | (df_rev < (low))).sum(axis=0), columns=['Total of outliers'])
outliers['% equivalent'] = round(outliers['Total of outliers']*100/len(df), 3)
outliers
Out[124]:
In [129]:
# Here i will replace the outliers of age by "higher acceptable range" due ot using mean the #of outliers increased from 59 to 131
q1 = df['age'].quantile(0.25) #first quartile value
q3 = df['age'].quantile(0.75) # third quartile value
iqr = q3-q1 #Interquartile range
low = q1-1.5*iqr #acceptable range
high = q3+1.5*iqr #acceptable range
df_in = df.loc[(df['age'] >= low) & (df['age'] <= high)] # meeting the acceptable range
df_out = df.loc[(df['age'] < low) | (df['age'] > high)] # not meeting the acceptable range
print('Shape of df without outliers = ' ,df_in.shape)
print('# of outliers =', df_out.shape)
print('lower rage = ' , low ,'&', ' Hiher rage =' , high)
#imputing outlier values with mean value
df_out.age=high
#getting back the original shape of df
df_rev=pd.concat([df_in,df_out]) #concatenating both dfs to get the original shape
df_rev.shape
print('Shape of original data frame', '' ,df.shape) # original data frame
print('Shape new df ' , ' ' , df_rev.shape) #new df
In [130]:
## checking the outliers again
q1= df_rev.quantile(0.25)
q3= df_rev.quantile(0.75)
IQR = q3-q1
low = q1-1.5*IQR #acceptable range
high = q3+1.5*IQR #acceptable range
outliers = pd.DataFrame(((df_rev > (high)) | (df_rev < (low))).sum(axis=0), columns=['Total of outliers'])
outliers['% equivalent'] = round(outliers['Total of outliers']*100/len(df), 3)
outliers
Out[130]:
In [131]:
q1 = df_rev['superplastic'].quantile(0.25) #first quartile value
q3 = df_rev['superplastic'].quantile(0.75) # third quartile value
iqr = q3-q1 #Interquartile range
low = q1-1.5*iqr #acceptable range
high = q3+1.5*iqr #acceptable range
df_in = df_rev.loc[(df_rev['superplastic'] >= low) & (df_rev['superplastic'] <= high)] # meeting the acceptable range
df_out = df_rev.loc[(df_rev['superplastic'] < low) | (df_rev['superplastic'] > high)] # not meeting the acceptable range
superplastic_mean=int(df_in.superplastic.mean()) #finding the mean of the acceptable range
print('Superplastic_mean =' ,superplastic_mean)
print('Shape of df without outliers = ' ,df_in.shape)
print('# of outliers =', df_out.shape)
print('lower rage = ' , low ,'&', ' Hiher rage =' , high)
#imputing outlier values with mean value
df_out.superplastic= superplastic_mean
#getting back the original shape of df
df_rev=pd.concat([df_in,df_out]) #concatenating both dfs to get the original shape
df_rev.shape
print('Shape of original data frame', '' ,df_rev.shape) # original data frame
print('Shape new df ' , ' ' , df_rev.shape) #new df
In [132]:
## The code from above is repeated to check the outliers again
q1= df_rev.quantile(0.25)
q3= df_rev.quantile(0.75)
IQR = q3-q1
low = q1-1.5*IQR #acceptable range
high = q3+1.5*IQR #acceptable range
outliers = pd.DataFrame(((df_rev > (high)) | (df_rev < (low))).sum(axis=0), columns=['Total of outliers'])
outliers['% equivalent'] = round(outliers['Total of outliers']*100/len(df), 3)
outliers
Out[132]:
In the code below I will run the same as 2.3.b after outlier correction¶
In [133]:
X = df_rev.drop('strength', axis=1) # Seperating the target and the rest
Y = df_rev['strength']
##Split into training and test set
x_train, x_test,y_train, y_test = train_test_split(X, Y, test_size=0.3, random_state=1) # 1 is just any random seed number
print (x_train.shape, x_test.shape)
x_train.head() # this is to review the columns
Out[133]:
In [134]:
num_folds = 50
seed = 7
kfold = KFold(n_splits=num_folds, random_state=seed)
#Linear Regression
model = LinearRegression()
model.fit(x_train, y_train)
cv = cross_val_score(model, X, Y, cv=kfold)
mean = round(cv.mean(), 3)
std = round(cv.std(), 3)
results = pd.DataFrame({'Model':['LinearRegression'],
'score_training': round(model.score(x_train, y_train),3),
'score_test':round(model.score(x_test, y_test),3),
'k_fold_mean':mean,
'k_fold_std':std,
'95% confidence intervals': str(round(mean-(1.96*std),3)) + ' <-> ' + str(round(mean+(1.96*std),3))
})
results = results[['Model', 'score_training','score_test','k_fold_mean', 'k_fold_std', '95% confidence intervals']]
#Linear Regression with Polynomial features of degree 2
model= Pipeline ([('poly', PolynomialFeatures(degree=2)),('reg',LinearRegression())])
model.fit(x_train, y_train)
cv = cross_val_score(model, X, Y, cv=kfold)
mean = round(cv.mean(), 3)
std = round(cv.std(), 3)
tempresults = pd.DataFrame({'Model':['LinearRegression degree 2'],
'score_training': round(model.score(x_train, y_train),3),
'score_test':round(model.score(x_test, y_test),3),
'k_fold_mean':mean,
'k_fold_std':std,
'95% confidence intervals': str(round(mean-(1.96*std),3)) + ' <-> ' + str(round(mean+(1.96*std),3))
})
results = pd.concat([results, tempresults])
#Linear Regression with Polynomial features of degree 3
model= Pipeline ([('poly', PolynomialFeatures(degree=3)),('reg',LinearRegression())])
model.fit(x_train, y_train)
cv = cross_val_score(model, X, Y, cv=kfold)
mean = round(cv.mean(), 3)
std = round(cv.std(), 3)
tempresults = pd.DataFrame({'Model':['LinearRegression degree 3'],
'score_training': round(model.score(x_train, y_train),3),
'score_test':round(model.score(x_test, y_test),3),
'k_fold_mean':mean,
'k_fold_std':std,
'95% confidence intervals': str(round(mean-(1.96*std),3)) + ' <-> ' + str(round(mean+(1.96*std),3))
})
results = pd.concat([results, tempresults])
#dt = DecisionTree Regressor
model = DecisionTreeRegressor(random_state=42, max_depth=4) #scale=True)
model.fit(x_train, y_train)
cv = cross_val_score(model, X, Y, cv=kfold)
mean = round(cv.mean(), 3)
std = round(cv.std(), 3)
tempresults = pd.DataFrame({'Model':['DecisionTree Regressor'],
'score_training': round(model.score(x_train, y_train),3),
'score_test':round(model.score(x_test, y_test),3),
'k_fold_mean':mean,
'k_fold_std':std,
'95% confidence intervals': str(round(mean-(1.96*std),3)) + ' <-> ' + str(round(mean+(1.96*std),3))
})
results = pd.concat([results, tempresults])
#dt = DecisionTree Regressor degree 2
model= Pipeline ([('poly', PolynomialFeatures(degree=2)),('reg',DecisionTreeRegressor(random_state=7))])
model.fit(x_train, y_train)
cv = cross_val_score(model, X, Y, cv=kfold)
mean = round(cv.mean(), 3)
std = round(cv.std(), 3)
tempresults = pd.DataFrame({'Model':['DecisionTree Regressor degree 2'],
'score_training': round(model.score(x_train, y_train),3),
'score_test':round(model.score(x_test, y_test),3),
'k_fold_mean':mean,
'k_fold_std':std,
'95% confidence intervals': str(round(mean-(1.96*std),3)) + ' <-> ' + str(round(mean+(1.96*std),3))
})
results = pd.concat([results, tempresults])
# RandomForestRegressor
model = RandomForestRegressor(random_state=42, max_depth=4)
model.fit(x_train, y_train)
cv = cross_val_score(model, X, Y, cv=kfold)
mean = round(cv.mean(), 3)
std = round(cv.std(), 3)
tempresults = pd.DataFrame({'Model':['Random Forest Regressor'],
'score_training': round(model.score(x_train, y_train),3),
'score_test':round(model.score(x_test, y_test),3),
'k_fold_mean':mean,
'k_fold_std':std,
'95% confidence intervals': str(round(mean-(1.96*std),3)) + ' <-> ' + str(round(mean+(1.96*std),3))
})
results = pd.concat([results, tempresults])
#Lasso
model=Lasso(random_state=42)
model.fit(x_train, y_train)
cv = cross_val_score(model, X, Y, cv=kfold)
mean = round(cv.mean(), 3)
std = round(cv.std(), 3)
tempresults = pd.DataFrame({'Model':['Lasso'],
'score_training': round(model.score(x_train, y_train),3),
'score_test':round(model.score(x_test, y_test),3),
'k_fold_mean':mean,
'k_fold_std':std,
'95% confidence intervals': str(round(mean-(1.96*std),3)) + ' <-> ' + str(round(mean+(1.96*std),3))
})
results = pd.concat([results, tempresults])
# Lasso with polynomial features of degree 2
model= Pipeline ([('poly', PolynomialFeatures(degree=2)),('reg',Lasso(random_state=42))])
model.fit(x_train, y_train)
cv = cross_val_score(model, X, Y, cv=kfold)
mean = round(cv.mean(), 3)
std = round(cv.std(), 3)
tempresults = pd.DataFrame({'Model':['Lasso degree 2'],
'score_training': round(model.score(x_train, y_train),3),
'score_test':round(model.score(x_test, y_test),3),
'k_fold_mean':mean,
'k_fold_std':std,
'95% confidence intervals': str(round(mean-(1.96*std),3)) + ' <-> ' + str(round(mean+(1.96*std),3))
})
results = pd.concat([results, tempresults])
# Lasso with polynomial features of degree 3
model= Pipeline ([('poly', PolynomialFeatures(degree=3)),('reg',Lasso(random_state=42))])
model.fit(x_train, y_train)
cv = cross_val_score(model, X, Y, cv=kfold)
mean = round(cv.mean(), 3)
std = round(cv.std(), 3)
tempresults = pd.DataFrame({'Model':['Lasso degree 3'],
'score_training': round(model.score(x_train, y_train),3),
'score_test':round(model.score(x_test, y_test),3),
'k_fold_mean':mean,
'k_fold_std':std,
'95% confidence intervals': str(round(mean-(1.96*std),3)) + ' <-> ' + str(round(mean+(1.96*std),3))
})
results = pd.concat([results, tempresults])
# Ada boosting
model = AdaBoostRegressor(n_estimators = 100, learning_rate=0.1, random_state=22)
model.fit(x_train, y_train)
cv = cross_val_score(model, X, Y, cv=kfold)
mean = round(cv.mean(), 3)
std = round(cv.std(), 3)
tempresults = pd.DataFrame({'Model':['Ada boosting'],
'score_training': round(model.score(x_train, y_train),3),
'score_test':round(model.score(x_test, y_test),3),
'k_fold_mean':mean,
'k_fold_std':std,
'95% confidence intervals': str(round(mean-(1.96*std),3)) + ' <-> ' + str(round(mean+(1.96*std),3))
})
results = pd.concat([results, tempresults])
# Gradient boosting
model = GradientBoostingRegressor(random_state=42, max_depth=4)
model.fit(x_train, y_train)
cv = cross_val_score(model, X, Y, cv=kfold)
mean = round(cv.mean(), 3)
std = round(cv.std(), 3)
tempresults = pd.DataFrame({'Model':['Gradient boosting'],
'score_training': round(model.score(x_train, y_train),3),
'score_test':round(model.score(x_test, y_test),3),
'k_fold_mean':mean,
'k_fold_std':std,
'95% confidence intervals': str(round(mean-(1.96*std),3)) + ' <-> ' + str(round(mean+(1.96*std),3))
})
results = pd.concat([results, tempresults])
## The next lines are to select the best model based on the k_fold_mean and add it in the last row
tmp_best = results.sort_values(['k_fold_mean'], ascending=False).head(1)
tmp_best['Model'] = 'Best Model = ' + tmp_best['Model']
results = results.append(tmp_best, ignore_index=True)
## This is the table with the scoring result of the algorithm with all the data
results
Out[134]:
In [142]:
# these are the variable ('ash', 'coarseagg', 'fineagg') with less correlation observed during the EDA
# 'Slag' could be dropped and evaluate the model later on. In this project I will not try due to lack of time
print (' Original columns: ',df_rev.columns)
df_rev2=df_rev.drop(['ash', 'coarseagg', 'fineagg'],axis=1)
print (' New columns: ', df_rev2.columns)
In [143]:
X = df_rev2.drop('strength', axis=1) # Seperating the target and the rest
Y = df_rev2['strength']
##Split into training and test set
x_train, x_test,y_train, y_test = train_test_split(X, Y, test_size=0.3, random_state=1) # 1 is just any random seed number
print (x_train.shape, x_test.shape)
x_train.head() # this is to review the columns
Out[143]:
In [144]:
num_folds = 50
seed = 7
kfold = KFold(n_splits=num_folds, random_state=seed)
#Linear Regression
model = LinearRegression()
model.fit(x_train, y_train)
cv = cross_val_score(model, X, Y, cv=kfold)
mean = round(cv.mean(), 3)
std = round(cv.std(), 3)
results = pd.DataFrame({'Model':['LinearRegression'],
'score_training': round(model.score(x_train, y_train),3),
'score_test':round(model.score(x_test, y_test),3),
'k_fold_mean':mean,
'k_fold_std':std,
'95% confidence intervals': str(round(mean-(1.96*std),3)) + ' <-> ' + str(round(mean+(1.96*std),3))
})
results = results[['Model', 'score_training','score_test','k_fold_mean', 'k_fold_std', '95% confidence intervals']]
#Linear Regression with Polynomial features of degree 2
model= Pipeline ([('poly', PolynomialFeatures(degree=2)),('reg',LinearRegression())])
model.fit(x_train, y_train)
cv = cross_val_score(model, X, Y, cv=kfold)
mean = round(cv.mean(), 3)
std = round(cv.std(), 3)
tempresults = pd.DataFrame({'Model':['LinearRegression degree 2'],
'score_training': round(model.score(x_train, y_train),3),
'score_test':round(model.score(x_test, y_test),3),
'k_fold_mean':mean,
'k_fold_std':std,
'95% confidence intervals': str(round(mean-(1.96*std),3)) + ' <-> ' + str(round(mean+(1.96*std),3))
})
results = pd.concat([results, tempresults])
#Linear Regression with Polynomial features of degree 3
model= Pipeline ([('poly', PolynomialFeatures(degree=3)),('reg',LinearRegression())])
model.fit(x_train, y_train)
cv = cross_val_score(model, X, Y, cv=kfold)
mean = round(cv.mean(), 3)
std = round(cv.std(), 3)
tempresults = pd.DataFrame({'Model':['LinearRegression degree 3'],
'score_training': round(model.score(x_train, y_train),3),
'score_test':round(model.score(x_test, y_test),3),
'k_fold_mean':mean,
'k_fold_std':std,
'95% confidence intervals': str(round(mean-(1.96*std),3)) + ' <-> ' + str(round(mean+(1.96*std),3))
})
results = pd.concat([results, tempresults])
#dt = DecisionTree Regressor
model = DecisionTreeRegressor(random_state=42, max_depth=4) #scale=True)
model.fit(x_train, y_train)
cv = cross_val_score(model, X, Y, cv=kfold)
mean = round(cv.mean(), 3)
std = round(cv.std(), 3)
tempresults = pd.DataFrame({'Model':['DecisionTree Regressor'],
'score_training': round(model.score(x_train, y_train),3),
'score_test':round(model.score(x_test, y_test),3),
'k_fold_mean':mean,
'k_fold_std':std,
'95% confidence intervals': str(round(mean-(1.96*std),3)) + ' <-> ' + str(round(mean+(1.96*std),3))
})
results = pd.concat([results, tempresults])
#dt = DecisionTree Regressor degree 2
model= Pipeline ([('poly', PolynomialFeatures(degree=2)),('reg',DecisionTreeRegressor(random_state=7))])
model.fit(x_train, y_train)
cv = cross_val_score(model, X, Y, cv=kfold)
mean = round(cv.mean(), 3)
std = round(cv.std(), 3)
tempresults = pd.DataFrame({'Model':['DecisionTree Regressor degree 2'],
'score_training': round(model.score(x_train, y_train),3),
'score_test':round(model.score(x_test, y_test),3),
'k_fold_mean':mean,
'k_fold_std':std,
'95% confidence intervals': str(round(mean-(1.96*std),3)) + ' <-> ' + str(round(mean+(1.96*std),3))
})
results = pd.concat([results, tempresults])
# RandomForestRegressor
model = RandomForestRegressor(random_state=42, max_depth=4)
model.fit(x_train, y_train)
cv = cross_val_score(model, X, Y, cv=kfold)
mean = round(cv.mean(), 3)
std = round(cv.std(), 3)
tempresults = pd.DataFrame({'Model':['Random Forest Regressor'],
'score_training': round(model.score(x_train, y_train),3),
'score_test':round(model.score(x_test, y_test),3),
'k_fold_mean':mean,
'k_fold_std':std,
'95% confidence intervals': str(round(mean-(1.96*std),3)) + ' <-> ' + str(round(mean+(1.96*std),3))
})
results = pd.concat([results, tempresults])
#Lasso
model=Lasso(random_state=42)
model.fit(x_train, y_train)
cv = cross_val_score(model, X, Y, cv=kfold)
mean = round(cv.mean(), 3)
std = round(cv.std(), 3)
tempresults = pd.DataFrame({'Model':['Lasso'],
'score_training': round(model.score(x_train, y_train),3),
'score_test':round(model.score(x_test, y_test),3),
'k_fold_mean':mean,
'k_fold_std':std,
'95% confidence intervals': str(round(mean-(1.96*std),3)) + ' <-> ' + str(round(mean+(1.96*std),3))
})
results = pd.concat([results, tempresults])
# Lasso with polynomial features of degree 2
model= Pipeline ([('poly', PolynomialFeatures(degree=2)),('reg',Lasso(random_state=42))])
model.fit(x_train, y_train)
cv = cross_val_score(model, X, Y, cv=kfold)
mean = round(cv.mean(), 3)
std = round(cv.std(), 3)
tempresults = pd.DataFrame({'Model':['Lasso degree 2'],
'score_training': round(model.score(x_train, y_train),3),
'score_test':round(model.score(x_test, y_test),3),
'k_fold_mean':mean,
'k_fold_std':std,
'95% confidence intervals': str(round(mean-(1.96*std),3)) + ' <-> ' + str(round(mean+(1.96*std),3))
})
results = pd.concat([results, tempresults])
# Lasso with polynomial features of degree 3
model= Pipeline ([('poly', PolynomialFeatures(degree=3)),('reg',Lasso(random_state=42))])
model.fit(x_train, y_train)
cv = cross_val_score(model, X, Y, cv=kfold)
mean = round(cv.mean(), 3)
std = round(cv.std(), 3)
tempresults = pd.DataFrame({'Model':['Lasso degree 3'],
'score_training': round(model.score(x_train, y_train),3),
'score_test':round(model.score(x_test, y_test),3),
'k_fold_mean':mean,
'k_fold_std':std,
'95% confidence intervals': str(round(mean-(1.96*std),3)) + ' <-> ' + str(round(mean+(1.96*std),3))
})
results = pd.concat([results, tempresults])
# Ada boosting
model = AdaBoostRegressor(n_estimators = 100, learning_rate=0.1, random_state=22)
model.fit(x_train, y_train)
cv = cross_val_score(model, X, Y, cv=kfold)
mean = round(cv.mean(), 3)
std = round(cv.std(), 3)
tempresults = pd.DataFrame({'Model':['Ada boosting'],
'score_training': round(model.score(x_train, y_train),3),
'score_test':round(model.score(x_test, y_test),3),
'k_fold_mean':mean,
'k_fold_std':std,
'95% confidence intervals': str(round(mean-(1.96*std),3)) + ' <-> ' + str(round(mean+(1.96*std),3))
})
results = pd.concat([results, tempresults])
# Gradient boosting
model = GradientBoostingRegressor(random_state=42, max_depth=4)
model.fit(x_train, y_train)
cv = cross_val_score(model, X, Y, cv=kfold)
mean = round(cv.mean(), 3)
std = round(cv.std(), 3)
tempresults = pd.DataFrame({'Model':['Gradient boosting'],
'score_training': round(model.score(x_train, y_train),3),
'score_test':round(model.score(x_test, y_test),3),
'k_fold_mean':mean,
'k_fold_std':std,
'95% confidence intervals': str(round(mean-(1.96*std),3)) + ' <-> ' + str(round(mean+(1.96*std),3))
})
results = pd.concat([results, tempresults])
## The next lines are to select the best model based on the k_fold_mean and add it in the last row
tmp_best = results.sort_values(['k_fold_mean'], ascending=False).head(1)
tmp_best['Model'] = 'Best Model = ' + tmp_best['Model']
results = results.append(tmp_best, ignore_index=True)
## This is the table with the scoring result of the algorithm with all the data
results
Out[144]:
3. Create Models and Tuning it¶
### Gradient boosting is the model that best predict the strength of concrete based on the given dataset and the studied models
### In the following steps I will try to tune the parameters of Gradient boosting
### Since the drop of the variable didn't improved the model (insight #12) we will use the df_rev which is after correcting the outliers
In [145]:
X = df_rev.drop('strength', axis=1) # Seperating the target and the rest
Y = df_rev['strength']
##Split into training and test set
x_train, x_test,y_train, y_test = train_test_split(X, Y, test_size=0.3, random_state=1) # 1 is just any random seed number
print (x_train.shape, x_test.shape, y_train.shape, y_test.shape)
x_train.head() # this is to review the columns
Out[145]:
In [148]:
from sklearn.model_selection import RandomizedSearchCV
GradientBoostingRegressor
# Prepare parameter grid
# Please notice: due to the lack of time to complet the project I took the parameters configuration below from internet,
# I checked different source and they had similar parameters with similar values.
# In real life I would need to learn more about this algorithm and the use of each parameters.
parameters = {
'criterion': ['mse', 'mae', 'friedman_mse'],
'learning_rate': [0.05, 0.1, 0.15, 0.2],
'max_depth': [2, 3, 4, 5],
'max_features': ['sqrt', None],
'max_leaf_nodes': list(range(2, 10)),
'n_estimators': list(range(50, 500, 50)),
'subsample': [0.8, 0.9, 1.0]
}
In [150]:
rs = RandomizedSearchCV(estimator=GradientBoostingRegressor(random_state=42), param_distributions=parameters,
return_train_score= True, n_jobs=-1, verbose=2, cv = 10, n_iter=500)
rs.fit(x_train, y_train)
Out[150]:
In [151]:
mean = rs.best_score_
std = rs.cv_results_['mean_test_score'].std()
print(f"Mean training score: {rs.cv_results_['mean_train_score'].mean()}")
print(f"Mean validation score: {mean}")
print(f"Validation standard deviation: {std}")
print(f"95% confidence interval: {str(round(mean-(1.96*std),3)) + ' <-> ' + str(round(mean+(1.96*std),3))}")
print(f"Best parameters: {rs.best_params_}")
print(f"Test score: {rs.score(x_test, y_test)}")
4. Conclusion regarding the best algorithm.¶
The best model is Gradient boosting having a high and similar score for the training and test data
Using this model is possible to predict the strength accurately between 82% to 100% with 95% confidence.
- The codes used in the steps 2.3 a), b) and c) could be created in a loop and it would be more efficient. For this project a tried to do it without success. I would need more time to figure out how to organize the loop.
- The same apply for the outliers correction, it could be donw with a loop wich is more less time consuming but it need more practice to be able to create it.
- Dropping variable didn't improve the results, however only 1 set of variables were dropped in this project, more trials could be done in this matter.
- It was confirmed that correcting the outliers with the mean could actually create more outliers. Also the outliers corrections helped to improve the overfit of some models.
- In this project the pandas report was used to confirmed the observations from the EDA. It is a usefull tool that can be used in future projecst as well as in the real life work.
- This final version of this project will be kept as a template for the future in order to be able to solve some problems much faster.
- Along the process of solving this project different observations were highlighted (insights) and added in the index in order to find them quickly.